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UARS: An Introduction 


1) Upper Atmosphere Research Satellite 
•) Launched Sept. 1991 

•) In planning and development stages since late i 970's 
•) Measures temperature, chemical species, winds, solar 
inputs 


2) Similarities to EOS 

•) Data collected and processed at a Central Data 
Handling Facility. 

•) Data distributed via high speed network to Remote 
Analysis Computers at investigator sites 
•) Science Team (users) include instrument Pis and 
theoretical Pis. 


3) Differences 

•) UARS is a one platform mission 
•) Highly focused on upper atmosphere research 
•) Quantitative global measurements of atmospheric 
parameters ( as opposed to determination of spectral or 
spatial contrast, event counting, etc) 

•) No imagery 


UARS Validation Chronology 


1) Validation not recognized as a fundamental requirement at 
the outset of the program, (due to semantics, oversight??) 

2) A series of events in 1988 focused the need: 

A) Within the UARS team it became apparent that some 
additional structure was required to unify; 

•) Calibration 

•) Algorithm verification 

•) Error analysis 

•) Correlative measurements 

•) A priori knowledge: (climatology, theory, modeling) 

B) The release of the Ozone Trends Panel Report 

3) UARS Validation Working Group created 1989 

4) Validation Plan completed 1991 

5) Plan Implementation 1991-92 
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Present State of Knowledge 
of the Upper Atmosphere 1988: 
An Assessment Report 


R. T. Watson and Ozone Trends Panel, 

M. J. Prather and Ad Hoc Theory Panel, 
and M. J. Kurylo and NASA Panel for 
Data Evaluation 

NASA Office of Space Science and Applications 
Washington, D.C. 

NASA recognizes the need for timely international scientific assessments when important 
new information becomes available as has occurred since the last major international 
scientific assessment (WMO, 1986). Reports based on Nimbus 7 satellite Solar 
Backscattcr Ultraviolet (SBUV) and Total Ozone Mapping Spectrometer (TOMS) data 
claimed that large global dearases have occurred since 1979 in the total column of ozone 
(about 1 % per year) and in its concentration near 50 km altitude (about 3% per year). Data 
from the ground-based Dobson network also indicated that the total column content of 
ozone had decreased on a global scale significantly since 1979, although to a lesser extent 
than suggested by the satellite data. Further, there has been a significant amount of new 
research focussed on understanding the extent and cause of the depletion of ozone in the 
spring-time over the Antarctic. 


NASA 

National Aeronautic 
and Space Administr 

Scientific and Techi 
Information Oivisior 


NASA and the rest of the scientific community believed that it was im p erati ve to evaluate 
whether the Nimbus 7 satellite data had been analyred^jnectly, and if so,'^tf>er the 
reported decreases were due to natural causes such as a decrease in solar radiation (from 
solar maximum in 1979 to solar minimum in 1986), the 1982 volcanic eruption of El- 
Chichon, or the 1982 El-Nino event, or whether it was due to human activities such as the 
use of chlorofl uorocarbons (CFCs). Therefore, during the fall of 1986 NASA decided to 
coordinate and cosponsor with the Federal Aviation Administration (FAA), the National 
Oceanic and Atmospheric Administration (NOAA), the World Meteorological Organization 
(WMO), and the United Nations Environmental Program (UNEP) a major review of all 
ground-based and satellite ozone data. A panel (the Ozone Trends Panel) composed of 
eminent scientists from federal agencies, research institutions, private industry, and 
universities was selected. 


Zen And The Art Of Data Validation 

t 

1) What Is Data Validation? 

A) What it is not. 

i) Header information 

ii) Flags marking data anomalies 

iii) Limit checks 

iv) Verifying that the software didn't bomb 

v) Documentation 

( These are all Quality Assurance Issues.. Necessary but not 
Sufficient) 

B) Also, What it is not: Comparing Profiles With 
Someone Else. ( A component but not an end in itself ) 

C) What it might Be. ( The Process of Demonstrating that a 
Collection of Data Represents the Real Atmosphere Within a 
Quantifiable Uncertainty) 

D) What it always is. 

i) Overlooked in program planning. 

ii) Underestimated in terms of time, effort and 

resources required. 

iii) The most frustrating part of the mission. 

2) Why Is It Important? 

UARS is not measuring gnylhlng for the fjrsl time. It is 
adding to a cumulative base of knowledge (in some cases, 
extensive) and therefore must be compatible with existing 
and future sources of Information 


Evolution of UARS Data Validation Plan 

1 ) Identification of issues within the Validation Working Group. 

2) Mandatory requirement that eac h investigation team 

prepare a plan for their activities. - — 

3) Creation of a "Generic" plan outlirrir " 

4) Development of Investigator specific plan outlines. ^ 

5) Review of Investigator specific draft plans 

6) Investigator specific final plans - - 

7) Collection of all investigator plans into overall plan 



Pre/Post Launch Validation Activities 


1) Pre-launch: 

A) Formulate Plans 

B) Identify resource requirements 

C) Begin development of tools and procedures 

2) Post-launch: 

A) Organize into issue/parameter specific Validation Sub- 
Groups 

•) Temperature/Pressure/Altitude registration 

•) Trace gas concentration 

•) Winds/Dynamics 

•) Solar Measurements 

•) Data gridding/mapping procedures 

•) Energetic Particles 

B) Investigator teams work through their validation plans 

C) Report findings 

D) Take corrective actions as necessary (instrument 
operation, data processing) 


GENERIC P.I. DATA VALIDATION PLAN OUTLINE 

1.0 INTRODUCTION 

1.1 Brief Experiment Overview, Including Measurements to be Validated 
and Altitude Ranges 

1.2 Brief Validation Criteria 

1.3 Validation Approach 

- Approach to Level 1, 2, and 3 validation (e.g. validate most 
understood parameters first, e.g. temperature and least 
understood parameters last) 

2.0 DESCRIPTION OF EXPERIMENT PHYSICAL MODEL 

2.1 Instrument Concept and Basic Equations 

2.2 Forward Radiance Model 

- Radiative transfer 

- Numerical approximations 

- Physical constraints (e.g. line parameters summary, plus 
reference) 

2.3 Inversion Approach 

- Brief description of basic approach 

- Constraint methods 

- Numerical approximations 

- Use of a priori information 

3.0 DESCRIPTION OF INSTRUMENT CHARACTERIZATION AND CALIBRATION 

3.1 Accuracy and Stability 

- IFC, temperature effects, noise, scale, and bias error 
stability 

3.2 Spectral Response and Registrations 

3.3 Spatial Response 

- FOV 

- Off-axis rejection 

- Crosstalk 
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3.4 Pointing 

3.5 Electronics Response 

- Amplitude and phase 

- Crosstalk 

3.6 Data System Errors 

- Gain uncertainties 

- Digitization errors 

3.7 Suntnary of Uncertainties with References 

4.0 ERROR ANALYSIS 

4.1 Sensitivity to Errors In Instrument Model 

4.2 Sensitivity to Errors in Forward Radiance Model 

4.3 Sensitivity to Inversion Algorithm Errors, Including A Priori 
Assumptions 

4.4 Spacecraft Effects 

- Altitude 

- Attitude rates 

- Ephemeris 

4.5 Uncertainties Due to Data Transmission (e.g. altitude 
Interpolation, True to Earth to IAU) 

4.6 Estimate of Total Measurement Error 

5.0 PRE-LAUNCH ACTIVITIES 

5.1 Instrument P.I. Obligations 

5.1.1 Define post-launch Instrument verification procedures 

5.1.2 Creation and comparison of Level 3AL data 

- Sample test atmosphere for 3 days 

- Synthesize radiances with production algorithm and add 
errors 

- Perform retrievals 

- Translate to standard latitudes for comparison 


5.1.3 Identify and develop tools and methods which will expedite 
post-launch validation 

5.2 Theoretical P.I. Support 

- Contributions by theoretical P.I.'s that will afd data 
validation 


POST-LAUNCH ACTIVITIES 

6.1 Instrument P.I. Obligations - . 

6.1.1 Implement instrument verification procedure 

- Monitor calibration stability (e.g. scale factor, bias) 

- Verify spectral^ registration 

- Verify spatial response characteristics 

- Evaluate correlation of Instrument signals with orbital 
events such as (e.g. south Atlantic anomaly, other 
instrument turn-on events, terminator crossing) 

6.1.2 Update error analysis as necessary 

6.2 Theoretical P.I. Support 

- Contributions by theoretical P.I.'s that will aid data 
validation 

6.3 Intercomparisons 

6.3.1 Guidelines 

- Number of comparisons wfth correlative measurements, 
locations, times, coincidence criteria (time, space) 

6.3.2 Climatology . - 

6.3.3 Correlative measurements 

6.3.4 Other UARS measurements - * - - 

6.3.5 Theory and derived products 

6.3.6. Targets of opportunity (e.g. ATLAS, NDSC) 
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7.0 IMPLEMENTATION 

7.1 Detailed Schedule with Milestones 

- Completion of on-orbit Instrument verification In procedure 
plan 

- Completion of on-orbit Instrument verification In procedure 
plan 

- Completion of initial on-orbit Instrument verification 
procedures 

- Validation of Level 1 products 

- Validation of Level 2 products 

- Validation of Level 3 products 

7.2 Resource Requirements 

- Personnel and equipment 

- Funding 

- Other 



Lessons Learned 
(Or Should Have Been) 

1) Start Early: Should be part of initial program planning. 

2) Put in adequate resources to support the goals 

•) If you want fast results, expect to pay 

•) If you want to save money, expect to wait 

3) Maintain better coordination between Validation 
planning/implementation and Correlative Measurement 
programs. Make sure they really compliment each other. 

4) Test correlative measurements data flow and validation 
procedures/tools well before launch. 

5) Divide the work: 

•) Instrument Pis are often overworked before and 
immediately after launch. 

•) Theoretical Pis are often under-utilized during this 
period. 

6) Learn from the successes and mistakes of others: Be willing 
to adapt as time goes along. 

7) Be realistic: (HQ is much better at setting goals than in 
providing the means to reach them.) 


Implications for EOS 


1) Use UARS as a "Living Laboratory" in an attempt to identify: 

•) what works 
•) what doesn't 
•) how to do it better 
•) what is realistic 

2) Make sure Correlative Measurement programs are planned 
with validation requirements in mind. Make sure they have 
appropriate resources, lead time and coordination with 
EOS. 

3) Enlist the "user" community to lend a hand: How should the 
work be divided? 

A) Instrument Teams take the lead in: 

•) Calibration 

•) Error analysis 
•) Level-1 data products 
•) Level-2 data products 

B) EOS "Users" take the lead in: 

•) Validation program planning 
•) Working group coordination 

•) Correlative measurement liaison 
•) Level-3 data products 

4) Validation activities continue for the life of the program 

•) There is an initial large "impulse" of activities with/; - 
each launch 

•) There is an ongoing "maintenance" effort ’for the life 
of each instrument 
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EOS Project Science Office 
Data Product Validation Policy 


DRAFT 
March 31, 1992 



Data Prodact Validation Poller 
March 31 . 1992 


INTRODUCTION 

EOS is a planned 15 year, multiple instrument/platform in the mssion to 

Planet Earth (MTPE) program designed to monitor 1 nrnrilirts to derive 

Numerous users of EOS data will rely on accurate EOS data ^ducts to de, rive 
higher generation data products. These data products and * he ^suiting 
scientific analyses will serve to guide environmental and economic policy. 

The scientific community will rely on the veracity of the data products 
developed in part because of our validation policy for those products and in 
partp'theba’n pf the scientific reputation of the investigators who are 
responsible for those products. 

in past satellite-based scientific investigations, data product validation has 
encompassed* (1) quality control checks on raw data; (2) generat o 
community-consensus , pee^-reviewed algorithms that transfer* the ^diance or 
reflectance measurements obtained from sensors into geophys * 

comparison of data products derived fro. satellite measurements with 
data products independently derived through techniques from orbiting, 
airborne, and ground-based instruments. 


REQUIREMENTS LEVIED BY THE 1988 EOS ANNOUNCEMENT OF OPPORTUNITY 

validation of the data products is established by comparing data products vlth 
measurement values acquired by conventional measurement and ^lys 
approaches. This experiment validation must be included in^eCallbratlon 
Plan provided in the proposed Instrument Investigation. The Data Product 
Validation Plan must define the correlative measurements and in-orbit 
calibratlon P plan which establishes conformance to the ^S ProJ^t Data Product 
Validation strategy. The instrument observables usually will be ^^preted 
as physical parameters, and are represented as data products . 0 

the data products is established by comparing data products with those 
acquired by conventional measurement, analysis, and other approaches. 

Specifically, the Data Product Validation Plan at a minimum must include: 

m A description of independent measurement and analysis approaches to be 
used in experiment validation and how the validation data products are 
to be compared to the instrument-derived data products. 


(2) A description of how the calibrations of instruments used in the 
validation network will be compared to the calibration of the 
Instruments in space. 


(3) 

(4) 

(5) 


An estimate of the accuracy and precision required in the validation 
data products so that they will be useful for this investigation. 

An estimate of the frequency, duration, location, and any appropriate 
special observing conditions required for the data validation 
measurements • 

Description of EOS validation measurement programs and the relationship 
between EOS validation measurement requirements and supporting major 


D»t* Prodoct Validation Policy 
March 31. 1992 

national and international science field measurement campaigns, such as 
FIFE, GEWEX , TOGA, etc. 


EOS PROGRAM OFFICE DATA PRODUCTS VALIDATION DEFINITIONS AND POLICY 
According to the EOS program office an EOS ? at * 

considered to be validated when several criteria are chronolog y y 

algorithms i must e pass 1 preflighlT a algoritta validation ^view. as must the 

™l 0 rder!ved V data r produc?s°virt Products 

recognised 1 the ^Tf 

verification activities: (2) anticipates that ^°Y^i ^vermfs ion 
will not take place; or, (3) does not encourage that these verification 

activities take place. In fact, campaigns to compare EOS data with truth co- 
located 1 measurements are viewed by the EOS program office as - 
vehicle in broadening the scientific community's interest in EOS. The ma 
ramification of removing these activities from under the data prod 
validation umbrella is that correlative measurement campaigns are not planned 
to be funded by the EOS program. 

The EOS Program Office definition of data product validation forces instrument 
investigators to more fully understand their instruments and a J g0 ^ t J“® 
placing more importance on preflight calibration and characterization tests 
that represent instrumental flight operations, instrumental 

models and algorithm verification. It also prompts instrument investigators 
^d daia ^odJcers to examine more cloaely their criteria for either accepting 
or rejecting a particular data set. 


EOS PROJECT SCIENCE OFFICE POLICY 

While it is the policy of the EOS Project Science Office that the guidelines 
“eJtiffed in the Announcement of Opportunity are still useful, there are few 
funds available to do more than verify — via peer-reviewed processes the 
suitability of algorithms. A measurement-based algorithm verification process 
likely 1 will be over a rather limited time frame and for a 11m ted set of 

environmental conditions for most of the data pro ucts. after its 

determined is how to deal with short-comings in a given algorithm after its 
official acceptance, whether these short-comings are due to incomplete capture 
of knowledge or due to a change in environmental parameters. 



EOS PROJECT SCIENCE OFFICE 
CROSS-CALIBRATION PLAN 


draft 

March 31 , 1992 


CrosB-C«l ibration Plan 
H.rch 25 , 1992 


INTRODUCTION 

Synergistic use of EOS data requires that instruments produce 
compatible measurements, even when several 

used. The Project must develop a technique that congruous 

Level 1 data products when the instruments are calibrated by the 
individual sensor builder*. The approaches being developed to 
accomplish this are round- robin laboratory comparisons and 
exposure of instruments to a common source after final 
calibration but previous to sensor integration onto the fl g 
platform. 

In addition, there exists the perception that all instruments 
will degrade in orbit, each at its own characteristic rate By 
knowing how the instruments compare on the ground before l a ^ c |V 
the earliest in-orbit comparisons will assist in establishing how 
these instruments have changed during launch. The combination of 
the long-term data sets then can be used to improve our 
understanding of each of these data sets. Our primary approach 
to supplementing the individual instrument calibrations for 
accomplishing this requirement of EOS is described in the Cross 
calibration Plan. 


In some sense, absolute calibration is not required for this 
activity. In principle, stable and precise calibrations could be 
used to meet these objectives. Nevertheless, experience has 
demonstrated that absolute calibrations are the only reliable 
approaches for accomplishing stable and precise calibrations. 
Cross-calibration has been added to supplement instrument 
absolute calibrations as the approach to making congruent data 
sets. 


Cross-calibration was made an EOS baseline requirement as defined 
in the 1988 Announcement of Opportunity (AO). Each Instrument 
Investigation is required to allow for such activities. 

There are several pre-flight instrument calibration alternatives. 


(1) Bring all instruments to a single facility where final 
radiometric and geometric calibration will be validated. 
This might provide the best calibration, but it could be 
very expensive and establish delays in getting the flight 
instruments delivered. 


(2) Have a transportable system that will be carried to the 
location where the instruments are being calibrated. This 
transportable system would be used to validate the local 
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Crocs- Calibration Plan 
March 2$. 1992 


calibration system and assure more compatibility between 
systems. This offers many of the advantages of the first, 
and fewer of the disadvantages. - 

(3) Depend upon each instrument builder to provide the 
transfer of the calibration through analys is and testing 
traceable to NIST sources. This ap proach is now commonly 
used, and generally suffers from a lack of adequate 
documentation. The results depend very much upon the 
specific people performing the calibration and the project 
requirements. 


From a logistical standpoint a single calibration facility or set 
of sources could lead to difficulties in launch scheduling. One 
cannot calibrate instruments until they have been built. 
Calibration is done as the last activity before shipping for 
integration. The use of a single set of sources or a single 
facility could lead to real difficulties in meeting the launch 
schedule. Cross-calibrations before instrument delivery also 
interfere with normal Project-contractor management interfaces. 

Thermal detectors for satellite radiometry always should be 
calibrated in a vacuum. Therefore, vacuum calibration facilities 
should be the norm for calibration on most of the EOS 
instruments. Such a facility will need to accommodate any of the 
instruments, and certain benefits result if the facility is large 
enough to accommodate the entire EOS satellite. Sources for 
calibration will operate in a vacuum. The sources must be 
mounted precisely within the instrument f ield-of-view. 

For EOS, a calibration scheme has been proposed that consists of 
several portable radiometers, each optimized for a certain 
spectral region. It is proposed that these radiometers be used 
in each instrument manufacturer's facility for comparison of the 
instrument calibration source scales. We refer to these as 
portable or traveling radiometers. 

Great strides have been made in the past years in detector-based 
precision radiometry. For the visible portion of the spectrum 
quantum- response detectors are now available that have 
uncertainties on the order of 0.1%. There is a high probability 
that comparable accuracies can be achieved at wavelengths 
extending to 1500nm in the next several years. Thermal detectors 
operated at room temperature are accurate to 0.1% for high input 
power levels and 1% for lower power levels. Cryogenic 
radiometers are now available with uncertainties approaching 
0.025% at modest power levels. This technology can be used 
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Cross-Cal ibrat Ion Plan 
March 25, 1992 


directly in the construction of high-accuracy radiometric 
instruments or indirectly in the calibration of stable 
instruments. These technologies could be well-matched to 
verification of the manufacturer's calibration source scale. 

There are pre-launch plans for the careful cross-calibration of 
the various laboratory sources using portable spectroradiometers 
and for a final cross-calibration of the instruments themse ves 
at the spacecraft integration facility. 


ROUND-ROBIN CROSS -CAL I BRAT I ON 

During instrument construction, the prime means of comparison 
should be through the circulation of transfer radiometers. These 
would compare the working targets that are used in the 
calibration of individual instruments. The primary function of 
these detectors is to verify the calibration of sources that are 
used to calibrate EOS instruments with VIS/NIR channels (e.g., 
MODIS-N, ASTER, MISR) . There is no perceived need for the 
circulation of standards for spectral or spatial calibration, as 
these topics can be handled through the use of standard 
procedures . 

Radiometers used as transfer standard radiometers must be shown 
to be stable, and their use must be documented through an error 
budget analysis. 

The AM Observatory is scheduled to be launched in June, 1998, and 
instruments will be delivered beginning two years before launch. 
The cross-calibration radiometers must be available by the summer 
of 1994 to support two years of testing before the instrument 
delivery. 


CROSS -CAL I BRAT I ON AT INTEGRATOR'S SITE 

The primary objective of the cross-calibration at the 
integrator's site is to determine the instrument-to-instrument 
bias when each instrument is looking at a well-controlled 
radiation field. This approach can establish the responsivitity 
of one instrument to another, but may not be useful in setting 
the absolute calibration scale of any one of them. 
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Cro»»-C«l 1 br#t ion Fisa 
March 25. 1992 


The EOS cross-comparison setup must accommodate a variety of 
instrument fields of view and aperture sizes, as well as operate 
over the full 0.4 pm to 15.4 pm waveband. Only radiometric 
comparisons will be made. Absolute calibration of the instruments 
shall be performed by the instrument builders prior to cross 
comparison. The requirements for cold space view (i.e., 4K cold 
plate) are TBD for cross-comparison. - - - 

Cross -comparison will occur at the spacecraft integrator's site. 
The integrator must provide support for cross-comparisons in 
their integration and test flow procedures. It is not necessary 
to accomplish an observatory level (all instruments at once) 
cross-calibration, and most calibrations should be performed 
during thermal vacuum testing. The Panel recommendation that 
there be separate calibration sources for visible/near IR and 
thermal IR calibrations. For thermal IR the panel recommended a 
more extended source, not an integrating sphere. 

Problems of cross-calibration at the spacecraft integrator s 
facility include tight schedules, difficulty in developing well- 
characterized targets of an appropriate common aperture, and the 
problem of controlling the setup and surroundings. There must be 
adequate time and facilities for detailed functional testing in 
thermal vacuum. 


CONCLUSIONS 


[TBD] 
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